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(57)Abstract: M 
PROBLEM TO BE SOLVED: To provide an operation m 

recognition device and method, and an information I 

record medium, enabling simplification of processing 
contents, reduction of processing time and reduction of 
cost. 

SOLUTION: An optical flow (a moving vector) 
calculating part 24 computes the average value of 
configuration pixels in each frame. A shaping processing 
part 26 conducts normalization processing for the 
average value. An image pickup pattern creating 
processing part 32 conducts voting processing for 
assigning a dot corresponding to the normalized value 
to a concerned position in a designated space, and 
repeats the processing for the number of frames. The 
space is divided into a designated number, and an image 
pickup pattern representing to which dividing position 
the dot is assigned is created by an image pickup 
pattern creating processing part 32. A pattern 
comparing part 34 compares the image pickup pattern 
with a plurality of registered patterns which have been previously registered and selects the 
nearest register pattern as an authentication result 
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CLAIMS 



[Claim(s)] 

[Claim 1]An operation recognition device comprising: 

An imaging means which picturizes a recognition object thing contained in a predetermined 
imaging range. 

A changing condition calculating means which computes a changing condition of a picture 
picturized by said imaging means about each of two or more frames. 

An image picked up pattern creation processing means to create an image picked up pattern in 
which it is shown to which region division said changing condition which divided into a region 
division of a predetermined number space specified by a variable of two or more ingredients 
which specifies said changing condition, and was computed about each of two or more of said 
frames corresponds. 

A registration pattern storing means which stores two or more registration patterns 
corresponding to two or more registering object operations created in the same way as said 
image picked up pattern created by said image picked up pattern creation processing means, A 
pattern comparing means which compares said image picked up pattern created by said image 
picked up pattern creation processing means with said registration pattern beforehand stored in 
said registration pattern storing means, and chooses said registration pattern nearest to said 
image picked up pattern. 

[Claim 2]An operation recognition device, wherein said two or more frames are classified into 
two or more groups corresponding to a time series variation in claim 1, and said image picked up 
pattern creation processing means creates said image picked up pattern so that a 
correspondence relation with said two or more groups may be known. 

[Claim 3]An operation recognition device, wherein said changing condition computed by said 
changing condition calculating means in claim 1 or 2 is the average value of a movement vector 
calculated about each pixel which constitutes said frame. 

[Claim 4]An operation recognition device, wherein said changing condition computed by said 
changing condition calculating means in claim 1 or 2 is the inter-frame variation of a centroid 
position of said recognition object thing. 

[Claim 5]An operation recognition device, wherein said change~of-state calculating means 
performs processing normalized so that the range of dispersion in said changing condition 
corresponding to each of two or more of said frames may become fixed in either of claims 1-4. 
[Claim 6]An operation recognition device, wherein said registration pattern stored in said 
registration pattern storing means supports the same registering object operation of multiple 
times in either of claims 1-5. 

[Claim 7]An operation recognition device whenever one changing condition of said frame after 
another is computed by said changing condition calculating means in either of claims 1-6, 
wherein said image picked up pattern creation processing means creates said image picked up 
pattern using a changing condition of said frame of a prescribed number before it 
[Claim 8]An operation recognition device when the m or more same selected results are 
included in claim 7 in a selected result of n batch continuously created by said image picked up 
pattern creation processing means, wherein this m or more selected result is used for said 
pattern comparing means. 



[Claim 9]An operation recognition device, wherein matching of said changing condition of a 
frame of 1 about said image picked up pattern or said registration pattern is performed to said 
two or more region divisions in either of claims 1-8. 

[Claim 10]An operation recognition device after choosing said registration pattern of 1 as 
activity of said recognition object thing in either of claims 1-9, wherein said pattern comparing 
means does not perform prescribed period selection operation. 

[Claim 11]A changing condition calculating means which computes a changing condition of a 
picture picturized by imaging means characterized by comprising the following about each of 
two or more frames, Space specified by a variable of two or more ingredients which specifies 
said changing condition is divided into a region division of a predetermined number, An image 
picked up pattern creation processing means to create an image picked up pattern in which it is 
shown to which region division said changing condition computed about each of two or more of 
said frames corresponds, A registration pattern storing means which stores two or more 
registration patterns corresponding to two or more registering object operations created in the 
same way as said image picked up pattern created by said image picked up pattern creation 
processing means, An operation recognition method of an operation recognition device provided 
with a pattern comparing means which compares said image picked up pattern created by said 
image picked up pattern creation processing means with said registration pattern beforehand 
stored in said registration pattern storing means, and chooses said registration pattern nearest 
to said image picked up pattern. 

A step which computes said changing condition by said changing condition calculating means. 
A step which creates said image picked up pattern by said image picked up pattern creation 
processing means, and a step which chooses said registration pattern of 1 by comparing said 
image picked up pattern with said registration pattern by said pattern comparing means. 

[Claim 12]An operation recognition method, wherein said two or more frames are classified into 
two or more groups corresponding to a time series variation in claim 11, and said image picked 
up pattern creation processing means creates said image picked up pattern so that a 
correspondence relation with said two or more groups may be known. 

[Claim 13]An operation recognition method, wherein said changing condition computed by said 
changing condition calculating means in claim 11 or 12 is the average value of a movement 
vector calculated about each pixel which constitutes said frame. 

[Claim 14] An operation recognition method, wherein said changing condition computed by said 
changing condition calculating means in claim 11 or 12 is the inter-frame variation of a centroid 
position of said recognition object thing. 

[Claim 15]An operation recognition method, wherein said change-of-state calculating means 
performs processing normalized so that the range of dispersion in said changing condition 
corresponding to each of two or more of said frames may become fixed in either of claims 11- 
14. 

[Claim 16]An operation recognition method, wherein said registration pattern stored in said 
registration pattern storing means supports the same registering object operation of multiple 
times in either of claims 11-15. 

[Claim 17]An operation recognition method whenever one changing condition of said frame after 
another is computed by said changing condition calculating means in either of claims 11-16, 
wherein said image picked up pattern creation processing means creates said image picked up 
pattern using a changing condition of said frame of a prescribed number before it 
[Claim 18]An operation recognition method when the m or more same selected results are 
included in claim 17 in a selected result of n batch continuously created by said image picked 
up pattern creation processing means, wherein this m or more selected result is used for said 
pattern comparing means. 

[Claim 19]An operation recognition method, wherein matching of said changing condition of a 
frame of 1 about said image picked up pattern or said registration pattern is performed to said 
two or more region divisions in either of claims 11-18. 

[Claim 20]An operation recognition method after choosing said registration pattern of 1 as 
activity of said recognition object thing in either of claims 11-19, wherein said pattern 
comparing means does not perform prescribed period selection operation. 



[Claim 21]A step which computes said changing condition by said changing condition calculating 
means to an operation recognition treating part characterized by comprising the following, An 
information recording medium which stored a program which performs a step which creates said 
image picked up pattern by said image picked up pattern creation processing means, and a step 
which chooses said registration pattern of 1 by comparing said image picked up pattern with 
said registration pattern by said pattern comparing means. 

A changing condition calculating means which computes a changing condition of a picture 
picturized by imaging means about two or more frames. 

An image picked up pattern creation processing means to create an image picked up pattern in 
which it is shown to which region division said changing condition which divided this space into 
a region division of a predetermined number while assuming space specified by a variable of two 
or more ingredients which specifies said changing condition, and was computed about each of 
two or more of said frames corresponds. 

A registration pattern storing means which stores two or more registration patterns 
corresponding to two or more registering object operations created in the same way as said 
image picked up pattern created by said image picked up pattern creation processing means. 
A pattern comparing means which compares said image picked up pattern created by said image 
picked up pattern creation processing means with said registration pattern beforehand stored in 
said registration pattern storing means, and chooses said registration pattern nearest to said 
image picked up pattern. 
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DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the identification unit of operation, method, and 
information recording medium which identify the contents of operation of a substance to be 
detected. 
[0002] 

[Description of the Prior Art]A motion of people's hand, the head, etc. is picturized with a 
camera, and various kinds of art of recognizing the contents of the motion based on the image 
pick-up result is known from the former. For example, the technique of Hidden Markov Model 
(HMM) is used, or the conventional technology which uses the technique of dynamic 
programming (DP) is known. By recognizing people's hand and the contents of the motion of the 
head using these techniques, it becomes possible to build the system etc. to which various 
kinds of operations corresponding to this recognition result are made to perform. 
[0003] 

[Problem(s) to be Solved by the Invention]By the way, the conventional method using Hidden 
Markov Model, dynamic programming, etc. which were mentioned above, Since [ this ] there was 
a problem that processing was complicated and processing time started, If it is going to choose 
the pattern of 1 corresponding to the photoed motion from many operation patterns registered 
beforehand, The highly efficient processing unit which can perform complicated processing for a 
short time is needed, a device becomes expensive, and it cannot apply to the system by which 
the manufacturing cost etc. are held down. 

[0004]this invention is created in view of such a point, and comes out. The purpose is simple for 
the contents and there are in providing the identification unit of operation, method, and 
information recording medium which can lower shortening and apparatus cost of processing 
time. 

[0005] 

[Means for Solving the Problem]In order to solve a technical problem mentioned above, an 
operation recognition device of this invention, An imaging means which picturizes a recognition 
object thing contained in a predetermined imaging range, and a changing condition calculating 
means which computes a changing condition of a picture picturized by imaging means about 
each of two or more frames, Space specified by a variable of two or more ingredients which 
specifies a changing condition is divided into a region division of a predetermined number, An 
image picked up pattern creation processing means to create an image picked up pattern in 
which it is shown to which region division a changing condition computed about each of two or 
more frames corresponds, A registration pattern storing means which stores two or more 
registration patterns corresponding to two or more registering object operations created in the 
same way as an image picked up pattern created by an image picked up pattern creation 
processing means, An image picked up pattern created by an image picked up pattern creation 
processing means is compared with a registration pattern beforehand stored in a registration 
pattern storing means, and it has a pattern comparing means which chooses a registration 
pattern nearest to an image picked up pattern. 

[0006]A changing condition calculating means in which an operation recognition method of this 



invention computes a changing condition of a picture picturized by imaging means about two or 
more frames, This space is divided into a region division of a predetermined number while 
assuming space specified by a variable of two or more ingredients which specifies a changing 
condition, An image picked up pattern creation processing means to create an image picked up 
pattern in which it is shown to which region division a changing condition computed about each 
of two or more frames corresponds, A registration pattern storing means which stores two or 
more registration patterns corresponding to two or more registering object operations created 
in the same way as an image picked up pattern created by an image picked up pattern creation 
processing means, An image picked up pattern created by an image picked up pattern creation 
processing means is compared with a registration pattern beforehand stored in a registration 
pattern storing means, A step which is performed in an operation recognition device provided 
with a pattern comparing means which chooses a registration pattern nearest to an image 
picked up pattern, and computes a changing condition by a changing condition calculating 
means, A step which creates an image picked up pattern by an image picked up pattern creation 
processing means, and a step which chooses a registration pattern of 1 by comparing an image 
picked up pattern with a registration pattern by a pattern comparing means are included. 
[0007]A changing condition calculating means in which an information recording medium of this 
invention computes a changing condition of a picture picturized by imaging means about two or 
more frames, This space is divided into a region division of a predetermined number while 
assuming space specified by a variable of two or more ingredients which specifies a changing 
condition, An image picked up pattern creation processing means to create an image picked up 
pattern in which it is shown to which region division a changing condition computed about each 
of two or more frames corresponds, A registration pattern storing means which stores two or 
more registration patterns corresponding to two or more registering object operations created 
in the same way as an image picked up pattern created by an image picked up pattern creation 
processing means, An image picked up pattern created by an image picked up pattern creation 
processing means is compared with a registration pattern beforehand stored in a registration 
pattern storing means, A step which computes a changing condition by a changing condition 
calculating means to an operation recognition treating part provided with a pattern comparing 
means which chooses a registration pattern nearest to an image picked up pattern, A program 
which performs a step which creates an image picked up pattern by an image picked up pattern 
creation processing means, and a step which chooses a registration pattern of 1 by comparing 
an image picked up pattern with a registration pattern by a pattern comparing means is stored. 
[0008]ln order to recognize activity which changes over a multiple frame since a changing 
condition of 1 of a picture is computed based on an image pick-up result for every frame, A 
changing condition for a frame number can be computed, a simple image picked up pattern can 
be created, activity can be recognized only by comparing with a registration pattern, the 
contents of processing are easy and shortening of processing time and reduction of apparatus 
cost are attained. 

[0009]Two or more frames mentioned above are classified into two or more groups 
corresponding to a time series variation, and, as for an image picked up pattern creation 
processing means, it is desirable to create an image picked up pattern so that a 
correspondence relation with two or more groups may be known. Thereby, recognition 
processing which took in a time factor of a series of recognition object operations becomes 
possible. 

[0010]As for a changing condition computed by changing condition calculating means mentioned 
above, it is desirable that it is the average value of a movement vector calculated about each 
pixel which constitutes a frame. Or as for a changing condition computed by changing condition 
calculating means mentioned above, it is desirable that it is the inter-frame variation of a 
centroid position of a recognition object thing. Since a changing condition corresponding to the 
whole picture for one picturized frame can be known by this, recognition of activity independent 
of a position in an imaging range is attained. 

[0011]As for a change-of-state calculating means mentioned above, it is desirable to perform 
processing normalized so that the range of dispersion in each corresponding changing condition 
of two or more frames may become fixed. Thereby, even if it is a case where sizes of operation 



of a recognition object thing differ, it becomes possible to recognize as the same activity 
regardless of the size. 

[0012]As for a registration pattern stored in a registration pattern storing means mentioned 
above, it is desirable to deal with the same registering object operation of multiple times. In 
order to create a registration pattern of 1, by repeating the same operation of multiple times 
and performing it, ambiguous recognition operation in consideration of a uncertainty of operation 
of people, etc. is realizable. 

[0013]As for an image picked up pattern creation processing means mentioned above, whenever 
one changing condition of a frame after another is computed by changing condition calculating 
means, it is desirable to create an image picked up pattern using a changing condition of a 
frame of a prescribed number before it Since this is enabled to recognize activity one after 
another continuously, specific operation (spotting operation) performed before carrying out 
recognition object operation becomes unnecessary. 

[0014]As for a pattern comparing means mentioned above, when the m or more same selected 
results are included in a selected result of n batch continuously created by an image picked up 
pattern creation processing means, it is desirable to adopt this m or more selected result. It 
enables this to reduce erroneous recognition generated suddenly. 

[0015]As for matching of a changing condition of a frame of 1 about an image picked up pattern 
or a registration pattern mentioned above, it is desirable to be carried out to two or more region 
divisions. Since an effect of adding a spatial shading off arises by this when a changing condition 
of 1 is computed, erroneous recognition produced and carried out can be prevented by 
assigning a changing condition to a region division of 1 by force. 

[0016]After a pattern comparing means mentioned above chooses a registration pattern of 1 as 
activity of a recognition object thing, it is desirable not to perform prescribed period selection 
operation. Since it is avoidable to carry out recognition processing to unprepared operation at 
the time of recognition object operation changing by this, etc., erroneous recognition can be 
prevented. 
[0017] 

[Embodiment of the Invention]Hereafter, the operation recognition device of one embodiment 
which applied this invention is explained in detail, referring to drawings. 

[A 1st embodiment] Drawing 1 is a figure showing the composition of the sensing device of a 
1st embodiment of operation. The operation recognition device shown in drawing 1 is 
constituted including the camera 10, the operation recognition treating part 20, and the 
information recording medium 40. 

[0018]The camera 10 is for picturizing various kinds of tools (for example, stick) which an 
operation recognition candidate's body contained in an imaging range and this operation 
recognition candidate possess with a predetermined time interval. The image data which 
picturized the recognition object thing is inputted from the camera 10, and the operation 
recognition treating part 20 recognizes the activity of a recognition object thing based on this 
image data. For this reason, the operation recognition treating part 20 is constituted including 
the image pick storage 22, the optical flow calculation part 24, the plastic surgery treating part 
26, the registration pattern creation treating part 28, the registration pattern storage 30, the 
image picked up pattern creation treating part 32, and the pattern comparing element 34. 
[0019]The image pick storage 22 stores the image data for every frame inputted with a 
predetermined time interval from the camera 10. The optical flow calculation part 24 calculates 
the optical flow for every predetermined infinitesimal area by reading the image data 
corresponding to two continuous frames from the image pick storage 22. Here, an optical flow is 
the movement vector calculated for every infinitesimal area, and the value corresponding for 
every pixel corresponding to image data is calculated. Calculation of the one-set optical flow 
corresponding to the image data for these two frames is performed using each image data of 
this frame and the frame in front of one for every interval to which the image data for one 
frame is outputted from the camera 10. 

[002Q] Drawing 2 is a figure showing the example of an optical flow. Each division shown in 
drawing 2 shows each pixel which constitutes one frame, and the arrow drawn into it shows the 
movement vector which is the pixel. The field on which the arrow is not drawn shows the pixel 



without the size of 0 motion of a movement vector. As shown in drawing 2 . the movement 
vector for every pixel which constitutes one frame is calculated as an optical flow. 
[0021]The optical flow calculation part 24 calculates the average value of the one-set optical 
flow corresponding to the image data for two frames. For example, it is considered as the 
coordinates (n(x n , y n ) = 0, 1 and 2, — ) of each pixel shown in drawing 2 . The value (X ingredient 

and Y ingredient of a movement vector) of the optical flow of the pixel of these coordinates is 

made into Fx^ and Fy^, and average value Fx_ of each ingredient and Fy -W- are calculated. 

n n ave ave 

[0022]The plastic surgery treating part 26 performs plastic surgery processing which carries out 
the multiplication of the predetermined magnification LX and the LY to the average value 
calculated by the optical flow calculation part 24 so that distribution of the average value of the 
one-set optical flow calculated about each of the frame of the predetermined number used as a 
recognition object may spread in the whole predetermined range. 

[0023]When a predetermined number is picturized for the operation for registration pattern 
creation by a frame with the camera 10 and the value (input value) after the plastic surgery 
corresponding to each of these frames is outputted from the plastic surgery treating part 26, 
the registration pattern creation treating part 28, The registration pattern which plotted the X 

ingredient Px (=Fx o xLx) and the Y ingredient Py ( = Fy aw xLy) of each input value in the square 

ave ave 

type region on a predetermined flat surface is created. 

[0024] Drawing 3 is a figure showing the example of the registration pattern created by the 
registration pattern creation treating part 28. In drawing 3 , Px corresponds to a horizontal axis, 
Py supports the vertical axis, respectively, and one point supports the input value for every 
frame. In the example shown in drawing 3 , since 12 points are included in the predetermined 
square type region, the input value of 12 pieces, i.e., the information corresponding to operation 
for 12 frames, will be included. 

[0025]In this square type region, it is divided horizontally n1, and is divided perpendicularly n2, 
and the n1xn2 piece region division is set up for the whole. According to this embodiment, one 
kind of registration pattern is set up by the number of the points plotted by each region division. 
In this specification, the processing which plots a point to an applicable region division is called 
a "voting process." 

[0026]The registration pattern storage 30 stores two or more registration patterns 
corresponding to each of two or more kinds of operations created by the registration pattern 
creation treating part 28. As mentioned above, the information which shows the number of the 
points included in each region division shown in drawing 3 is included in each registration 
pattern. 

[0027]When the input value after the plastic surgery corresponding to these frames is inputted 
from the plastic surgery treating part 26 by which the predetermined number was picturized for 
recognition object operation by the frame with the camera 10, the image picked up pattern 
creation treating part 32, The image picked up pattern which plotted the X ingredient Px and 
the Y ingredient Py of each input value in the square type region on a predetermined flat 
surface is created. This specification shall explain by calling the result of having performed the 
voting process based on the image data of recognition object operation an "image picked up 
pattern." 

[0028]The creation point of this image picked up pattern is the same as that [ creation ] of the 
registration pattern mentioned above, and processing which asks for the number of the points 
for every region division shown in drawing 3 about operation used as a recognition object is 
performed. As shown in drawing 3 , it is divided horizontally n1, and is divided perpendicularly n2, 
and one image picked up pattern is created by counting the number of the points plotted by 
each region division. 

[0029]The pattern comparing element 34 compares the registration pattern corresponding to 
each of two or more kinds of operations stored in the registration pattern storage 30 with the 
image picked up pattern corresponding to the recognition object operation created by the image 
picked up pattern creation treating part 32, and extracts the registration pattern nearest to an 
image picked up pattern. For example, the difference of the number of points is searched for for 
every region division in the square type region shown in drawing 3 , and this comparison is 



further performed based on the size of the accumulated (the accumulated of the absolute value 
of a difference may only be used) about all the region divisions in the square of this difference. 
That is, in order to show that an image picked up pattern and this registration pattern of 0 
correspond thoroughly in this accumulated, when such a registration pattern exists, this 
registration pattern is chosen as a comparison result. When the registration pattern in which 
accumulated was set to 0 does not exist, a registration pattern with the smallest accumulated 
is chosen as a comparison result. 

[0030] Drawing 4 is a figure showing the outline of the comparison operations by the pattern 
comparing element 34. In drawing 4 , three kinds of patterns A, B, and C shall be beforehand 
registered as a registration pattern. If image picked up pattern D corresponding to the newest 
St frame is inputted, the pattern comparing element 34 will compare each of the registration 
pattern A, B, and C with image picked up pattern D, and will choose the registration pattern A 
with the smallest accumulated that squared the difference of the number of the points of each 
region division. 

[0031 ]The information recording medium 40 is because the data of a program required for 
implementation of the recognition processing performed by the operation recognition treating 
^part 20 or a registration pattern is stored. This information recording medium 40 is constituted 
using semiconductor memory, such as optical disk type storage media, such as CD (compact 
disk), ROM, RAM, or a hard disk drive. 

[0032]For example, when ROM and RAM which stored the program for a series of operation 
recognition as the information recording medium 40 are considered. The optical flow calculation 
part 24 contained in the operation recognition treating part 20, the plastic surgery treating part 
26, the registration pattern creation treating part 28, the image picked up pattern creation 
treating part 32, and the pattern comparing element 34, It can realize by executing this program 
by CPU, and RAM which stores image data and registration pattern information can realize the 
taken image storage 22 and registration pattern storage 30. 

[0033]In the camera 10 mentioned above, to an imaging means, the optical flow calculation part 
24 and the plastic surgery treating part 26 to a changing condition calculating means. The image 
picked up pattern creation treating part 32 corresponds to an image picked up pattern creation 
processing means, the registration pattern storage 30 corresponds to a registration pattern 
storing means, and the pattern comparing element 34 corresponds to a pattern comparing 
means, respectively. 

[0034]The operation recognition device of this embodiment has such composition, and explains 
the operation below. Drawing 5 is a flow chart showing the rough operation procedures of the 
recognition processing of the activity by the operation recognition device of this embodiment If 
the picture for one frame containing a recognition object thing is picturized and acquired with 
the camera 10, the image pick storage 22 stores in the image pick storage 22 the image data 
acquired by this image pick-up (Step 100). 

[0035]Next, the optical flow calculation part 24 computes the one-set optical flow 
corresponding to the image data for two frames stored in the image pick storage 22, and its 
average value, Average value Fx_ of the lot corresponding to the image data of one frame 

used as a recognition object and Fy ave are outputted (Step 101). The plastic surgery treating 

part 26 performs plastic surgery processing which carries out the multiplication of the 
predetermined magnification LX and the LY to this average value, and outputs the input value 
Px of a lot and Py (Step 102). 

[0036]Next, the registration pattern creation treating part 28 judges whether register mode is 
specified (Step 103), and the image picked up pattern creation treating part 32 judges whether 
recognition mode is specified (Step 104). In the operation recognition device of this 
embodiment, before performing the image pick-up with the camera 10, either "register mode" or 
"recognition mode" shall be specified selectively. When "register mode" is specified, the 
registration pattern creation treating part 28 performs an affirmative judgment in the judgment 
of Step 103, next carries out the voting process corresponding to the image pick for one frame 
(Step 105). For example, when it is the registration pattern in which the whole square type 
region shown in drawing 3 was completed, processing which plots one point corresponding to 
this one frame, and assigns it to an applicable region division as a voting process corresponding 



to the image pick for one frame is performed. 

[0037]An end of the registration processing for one frame will judge whether a series of 
processings about one registration pattern creation ended the registration pattern creation 
treating part 28 (Step 106). If the time corresponding to one registration pattern, i.e., frame 
number St, shall be set up, when processing for the frame of this St individual will not be 
completed, A negative judgment is performed in the judgment of Step 106, it returns to Step 
100, and acquisition of the image data for one frame and the processing after storing operation 
are repeated. When processing for the frame of St individual is completed, an affirmative 
judgment is performed in the judgment of Step 106, next the registration pattern creation 
treating part 28 creates one registration pattern corresponding to the frame of St individual, and 
this is stored in the registration pattern storage 30 (Step 107). Then, it returns to Step 100 and 
acquisition of the image data for one frame and the processing after storing operation are 
repeated. Thus, registration processing about 1 or two or more registration patterns used as a 
recognition object is performed. 

[0038]On the other hand, when "recognition mode" is specified, the image picked up pattern 
creation treating part 32 performs an affirmative judgment in the judgment of Step 104, next the 
image picked up pattern creation treating part 32 carries out the voting process corresponding 
to the image pick for one frame (Step 108). In being the image picked up pattern of recognition 
object operation in which the whole square type region shown in drawing 3 was completed like 
the voting process which followed the registration pattern mentioned above, for example, 
Processing which plots one point corresponding to this one frame, and assigns it to an 
applicable region division as a voting process corresponding to the image pick for one frame is 
performed. 

[0039]An end of the voting process for one frame will judge whether the processing about a 
series of image picked up patterns corresponding to one recognition object operation ended the 
image picked up pattern creation treating part 32 (Step 109). When processing for the frame of 
St individual is not completed, a negative judgment is performed in the judgment of Step 109, it 
returns to Step 100, and acquisition of the image data for one frame and the processing after 
storing operation are repeated. When processing for the frame of St individual is completed, It is 
performed by the affirmative judgment in the judgment of Step 109, and next, the pattern 
comparing element 34, A series of image picked up patterns corresponding to recognition object 
operation are compared with each registration pattern stored in the registration pattern storage 
30, and a difference chooses the smallest registration pattern as a recognition result to 
recognition object operation (Step 110). 

[0040]In the operation recognition device of this embodiment, lot calculation of the motion 
ingredient of a recognition object thing is carried out for each [ which was picturized with the 
camera 10 ] frame of every, The comparison operations to two or more comparison patterns 
registered beforehand are performed using the image picked up pattern created by judging the 
region division corresponding to this motion ingredient. Thus, in order to recognize the activity 
which changes over a multiple frame since the average value of an optical flow is computed 
based on the image pick-up result for every frame, Each average value for a frame number can 
be computed, a simple image picked up pattern can be created, activity can be recognized only 
by comparing with a registration pattern, the contents of processing are easy and shortening of 
processing time and reduction of apparatus cost are attained. 

[0041]By the way, in the explanation of this embodiment mentioned above, in order to create a 
registration pattern, were made to carry out a series of operations only once, but In the case 
where a motion of people's hand and the head is recognized etc., the recognition technique 
which is difficult for reproducing the same motion correctly each time even if the same person 
performs the same motion, and permits a certain amount of ambiguity may be desired. 
[0042]It is, even if it adopts the technique 1 or the technique 2 shown below as a technique in 
the case of permitting such ambiguity, and it is **. 

(Technique 1) When creating a registration pattern, the same operation is repeated two or more 
times, and a registration pattern is created. For example, after having considered the case 
where the same operation was repeated 3 times the registration pattern shown in drawing 3 
corresponding to the 1st operation is created, corresponding to the 2nd time and the 3rd 



operation, it is made to perform a voting process to the same registration pattern in piles. If 
three operations are completely the same, the number of the points included in each region 
division shown in drawin g 3 will increase 3 times correctly, but when operation shows dispersion, 
predetermined breadth arises in the region division where each point is contained. Therefore, 
when the comparison operations to recognition object operation are performed next, a 
possibility of being chosen becomes high in being close to either of three register operation 
mentioned above. 

[0043](Technique 2) When creating a registration pattern, in the voting process at the time of 
creating the image picked up pattern corresponding to recognition object operation, it is made 
to perform spatial shading-off processing. Drawing 6 is a figure showing the example of the 
voting process at the time of performing spatial shading-off processing. The case where three 
voting processes corresponding to three frames are performed is shown by the example shown 
in drawing 6 . It is a center at the time of the center of each circle which has predetermined 
breadth casting its vote, and the circle which has a predetermined radius around these centers 
is drawn. Density is thin, so that this circle has so deep density that it approaches a center and 
it approaches on the outskirts (in drawing 6 , density is so deep that the interval of hatching is 
narrow, and it is considered as what has thin density, so that an interval becomes large). In 
consideration of the area and density of a circle which are contained in each region division, the 
ballot results of each region division are determined. That is, although vote was performed to 
any 1 region division in the example shown in drawing 3 by one vote corresponding to one 
frame, vote is performed to two or more region divisions in the example shown in drawing 6 by 
one vote corresponding to one frame. Some examples can be considered about the method of 
the vote to each region division. For example, the case where the result of having carried out 
the multiplication of the area and density correctly for each [ a part of circle is contained ] 
region division of every is made into ballot results, and the case where make the predetermined 
numerical value a correspond to the region division where the center of a circle is contained, 
and the predetermined numerical value b «a) is made to correspond to each region division 
where a part of other circle is contained can be considered. 

[0044] Drawing 7 is a figure showing other examples of the voting process at the time of 
performing spatial shading-off processing. In the example shown in drawing 7 , the whole 
quadrangle to which hatching was given shows 1 time of the ballot results corresponding to one 
frame. That is, to having plotted one point to the region division of 1 in one voting process, in 
one voting process, three points are plotted to the region division where a voting value is 
included, and one point is plotted to each eight region division of the circumference by the 
example shown in drawing 7 in the example shown in drawing 3 . Even if it does in this way, it 
becomes possible to obtain a result equivalent to the case where a voting process is performed 
using the circle which has the predetermined radius shown in drawing 6 . 
[0045]Since various motions which are not directly related to a registration pattern are 
picturized with the camera 10 when recognition object operation changes, the case where these 
various motions serve as a noise and appear can be considered. Drawing 8 is a figure showing 
the example of the method of removing the noise contained in a recognition result As shown in 
drawing 8 , whenever the image data of each frame is outputted from the camera 10, comparison 
processing using the St frame just before this frame is included is carried out, and the 
registration pattern nearest to an image picked up pattern is extracted. When its attention is 
paid to 8 times (=n) of the registration patterns which repeated this extracting processing for 
every frame, for example, were extracted immediately before, and the same registration pattern 
is contained more than 6 times (=m), this registration pattern is determined as a recognition 
result In the example shown in drawing 8 , as 8 times of latest recognition results, since 6 times 
and the pattern Y are extracted twice, the pattern X is determined as a final recognition result 
[ pattern / X ] at this time. Thus, reduction of the noise contained in a recognition result is 
attained. In the example shown in drawing 8 , the pattern Y extracted twice is removed as a 
noise. 

[0046][A 2nd embodiment] According to a 1st embodiment mentioned above, since its vote is 
cast in the way same about the total range of recognition object operation, the time factor of 
each operation included in recognition object operation is not taken into consideration. 



Therefore, when there is recognition object operation which consists of the operation A and the 
operation B, and performing previously the case where operation A is performed previously, and 
operation B, it becomes the same recognition result. On the other hand, in the operation 
recognition device of a 2nd embodiment, recognition processing in consideration of the time 
factor of each operation included in recognition object operation is made possible by doing n 
division of the frame of St individual contained in recognition object operation, and performing a 
voting process for every St/n frame. The basic constitution of the operation recognition device 
of a 2nd embodiment is the same as the operation recognition device of a 1st embodiment 
shown in drawing 1 , and detailed explanation is omitted. 

[0047] Drawing 9 is a figure showing the creation point of the registration pattern of a 2nd 
embodiment, and an image picked up pattern. In drawing 9 , n trichotomy (for example, 
quadrisection) of the frame of St individual contained in recognition object operation is done, 
and the ballot results for every three St/n are put in order along with the time-axis T. 
[0048] Drawing 10 is a figure showing the outline of the comparison operations by the pattern 
comparing element 34. In drawing 10 , three kinds of patterns A, B, and C in which the time 
factor was taken into consideration shall be beforehand registered as a registration pattern. If 
image picked up pattern D corresponding to the newest St frame is inputted, the pattern 
comparing element 34 will compare each of the registration pattern A, B, and C with image 
picked up pattern D, and will choose the registration pattern A with the smallest accumulated 
that squared the difference of the number of the points of each region division. 
[0049]Thus, n trichotomy of the frame of St individual corresponding to recognition object 
operation is carried out, and it becomes possible to recognize the activity which also took into 
consideration one-by-one time application about two or more operations which constitute 
recognition object operation by [ corresponding to each which was divided ] performing a voting 
process for every multiple frame. 

[0050]This invention is not limited to the above-mentioned embodiment, and various 
modification implementation is possible for it within the limits of the gist of this invention. For 
example, although each of the registration pattern and the image picked up pattern was created 
in the embodiment mentioned above using the frame of a prescribed number, When the duration 
time of the operation at the time of creating a registration pattern differs from the duration time 
of the operation at the time of creating an image picked up pattern, the duration time of the 
operation at the time of creating an image picked up pattern is shortened or extended by 
calculation, and it may be made to coincide duration time of operation. For example, when the 
value of the optical flow corresponding to the image pickup position of 1 is calculated as a 
discrete value corresponding to each frame, to perform interpolation processings (linear 
interpolation processing, spline interpolation processing, etc.) which used these discrete values, 
and what is necessary is just made to change a frame number. 

[0051]Although the embodiment mentioned above explained the operation in the case of 
recognizing the contents of recognition object operation continuously using the picture inputted 
from the camera 10, Since a noise is contained in a recognition result when recognition object 
operation changes, as mentioned above, in order to prevent this, after one recognition 
processing is completed and a recognition result is outputted, fixed time (for example, for 3 
seconds) recognition processing is suspended, and it may be made to suppress generating of 
the noise by erroneous recognition. 

[0052]Although the changing condition of the picture for every frame was detected in quest of 
the average value of an optical flow in the embodiment mentioned above, The center of gravity 
of a recognition object thing is searched for for every other method, for example, a frame, and 
after detecting the quantity which this centroid position moves by inter-frame, a normalizing 
process is performed and it may be made to perform a voting process. 

[0053] Drawing 11 is a figure showing the example in the case of searching for the center of 
gravity of the recognition object thing contained in a frame. In order to search for the center of 
gravity G of a recognition object thing as shown in drawing 1 1 . it is necessary to pretreat 
beforehand and to extract the outline R of a recognition object thing. For example, when a 
background color is set to white etc., by comparing the luminosity of each pixel with a 
predetermined threshold, only a recognition object thing can be extracted and it becomes 



possible to search for the center of gravity G about the pixel which exists in the inside of the 
outline R of this recognition object thing. Thus, it may be made to perform a voting process 
using the continuation 2 inter-frame difference (variation) of the center of gravity G searched 
for. 
[0054] 

[Effect of the lnvention]As mentioned above, in order to recognize the activity which changes 
over a multiple frame since the changing condition of 1 of a picture is computed based on the 
image pick-up result for every frame according to this invention, The changing condition for a 
frame number can be computed, a simple image picked up pattern can be created, activity can 
be recognized only by comparing with a registration pattern, the contents of processing are 
easy and shortening of processing time and reduction of apparatus cost are attained. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is a figure showing the composition of the sensing device of a 1st embodiment of 
operation. 

[Drawing 2]It is a figure showing the example of an optical flow. 

[Drawing 3] It is a figure showing the example of the registration pattern created by the 
registration pattern creation treating part 

[Drawing 4] It is a figure showing the outline of the comparison operations by a pattern 
comparing element. 

[Drawing 5]It is a flow chart showing the rough operation procedures of the recognition 
processing of the activity by the operation recognition device of this embodiment. 
[Drawing 6] It is a figure showing the example of the voting process at the time of performing 
spatial shading-off processing. 

[Drawing 7] It is a figure showing other examples of the voting process at the time of performing 
spatial shading-off processing. 

[Drawing 8] It is a figure showing the example of the method of removing the noise contained in 
a recognition result 

[Drawing 9] It is a figure showing the creation point of the registration pattern of a 2nd 
embodiment and an image picked up pattern. 

[Drawing 10] It is a figure showing the outline of the comparison operations by a pattern 
comparing element 

[Drawing 1 1] It is a figure showing the example in the case of using the variation of the center of 
gravity. 

[Description of Notations] 
10 Camera 

20 Operation recognition treating part 

22 Image pick storage 

24 Optical flow calculation part 

26 Plastic surgery treating part 

28 Registration pattern creation treating part 

30 Registration pattern storage 

32 Image picked up pattern creation treating part 

34 Pattern comparing element 
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so Xf77l 0 0EloT17U-A»OSif-^©i 
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>ft&®m$2 8ti, S t ficD^b-A^^f^^ 1 O 
OlS/^-ySML, cn^^/^->^Si53 
OlZ&m-tZ Uf7/1 0 7) o Xf7^1 
0 OIcM-dT 1 ^U-A^OM^x-^O^f. ts^a 

aiw^n^o 10 

[0 0 3 8] -75", rBW-t— FJ *<Jg^*tlTl^*« 
CFfiEJlLSSP 3 2 17 U-A^OflMSBfBJcWlS-r * 

S3!«ya*sisfi-ra (xfy^i os) a laifess 

[0 0 3 9] 1 7b-A#Oj8*fflItf«7t5 fc, » 

32^ 1 o©Bwt*»tt k:» 

3^*W£-rs Uf7^1 0 9) o S t!07l/-A» 

i7i/-A^iSf-^«#, tttrnfmnaaui 

6«DiBSnSo S tI®7l/-A»Offlltf» 

TL/cit^Cte. Xf l ^10 9©WS^Tt$flJ 30 

e*i:LTas?-rs (xf7^i 1 o) 0 
[0040] 4^flse>Bonfm^BTa. 1 

0fcJ;oT»«$n^7U~A«tEI»»Mf 

fiotl^c C coddle, 7l^-A«<Dffl^*tcS-r5 
^T*^f-r*;l/7o^O¥^I^»tti?ns/'c», » 

[004 1] ^c^t% ±M\,tc*nmmm<DmmT* 
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iE»tciRii:ib*«wa-r*ckttiiL<, &safto» 
tt^ *fF§f s B«¥ffi# as nz>m& t fe £ o 

[0 0 4 2] citDcfcaaSW^^fFSfsa'&O^ffii: 

ttff*3ia«03fi-rii^*#**i:, lHBOftfftW 
j£LTH3^LftB»^^*— >*^J«Stife*. 2 0 
BteAtf 3|sIB(0l(ifffc:»*UTrai:fia/^*— 
faTSSf«ia*fT5<fc5^-r«o 3H©llft3B<i<B- 
■Cfctitf, H3»c*Lfc*»««l*fc:**ti*jaoft3V 

[0043] (*»2) a»^*->*^dc-rsis, fe 

««ci*-raa»tt**assnSo -rfttrs. 03^ 
Lfc«T?«. 1 ^u-A^j&ra 1 laioaaR-ett, ^ 

rnfr— JtLTjaJRtfff tonfcW, IS 6 ic 

^tm^t. 1 7i/-Atcjjj£ts 1 @ofii7, 

o»*J«*K:»LTJBai«<fftotia. ft43. «»w«« 

coSffib «a) *i*^S*S«-&*^^L6nSo 
[0 0 4 4] i7ii, ^Wfti^LSa^tTo/c^ 

MTtt. A^fyWf^tl/-c0ftM^ 1 71/-A 

3^^bfc0ijT^i 1 ©c0«3S«yik:*5t^T-O»9J®« 

it 1 motsmwmizts^T. mmm^^n^mmm 
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[0 0 4 5] Sfc, aaB*ftft^«]0»to*«^U: 

S Q 0 8^t 1 0^6*7 U—AcoB 
«f-?W?h5«^ C<D7U— A*#tyfiMtl<D 10 

6 HI (=m) W±t$ntV^f^COl 
B^^— V*B«te*fcLT8feS^rSo 0 8tc^TffllT* 
It, m7B:<D8®(Dmffi!feMtLT^&~ ^X&6m* 

&s 0 H8Jc^Lrct»iTt±s 2 0jftffi^rn^^*— >y 

[0 0 4 6] BB 2 ^USSff^] ±38 L fcSS l OHfiSJB 

*ffoTl^*ft». B«3«Rftfffc#*ti**ft^«F 

ffB6^6a*BB«>M^*<a5ofcfc#k:. »ftA*$fc 
fr 3 *& i: »flF B *5fcK ff 5 »&(± n ttSMtSSI t ft 

«Btt*^$n«S tl07l/^n»|JU 30 
S t/n^U— A#Jctft»jaB|*ff5 Cfcfc:J:D. B3E 
^tt^^^Sns^ftffOBfMWSJR^^&tLfcBa 

m£M(DW*mf&tim 1 fc^Lfc* i <o^mmm(om^ 

[0 0 4 7] H9t±. S2 0HMJB«OB»^* — >;& 
.fctftWH*— >©fft*BSB**VfH'T*fcSo 0 9 1? 

»J (#Rfcf4»fiJ) StlTfctK S t/n3 7L/-AS 
[0 0 4 8] HlOli, — >it«SP3 4 tC£Sj±S> 

So 

[0 0 4 9] ccoj:^tc. il»fiffic»i£tss t so 
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So 

[0050] *awi±±B«iJgtt«ciBSsti* 

^pT^TfeSo PJ*tf. ±ifiUfcjB«»»T?tt, Bfi^ 

* - >fe cfc tf < * - xofhfhSBrSfflfto 7 u 
-A*Jfl^Tm£L;te#. a»/<*->*ff«Lfe|RO 

# 7 u- a testis l rmmm t l t §1-312 nrc t # tc N 
cn5oii«*»v^}iiiiijBi (ffittffiHfflffl^x^ 

[0051] £7c, ±aLfenaiBBaTj±, 5 1 0 
^6x**nsa«*fflv^TBW»«M&f^orts*a« 

WJcBM-rs^OftfttWWbfc^. ±3«bfcJ:5te: 

mmttmmiww k> mt> s istcsaiuem^ / ^ x*^s 

ns/c#>. cn^ESi-rSfcfete:, lHOBttJQ&tfft 
7LTBWS**«m**tifc*fc, -^«BBB 3 
»RB) BWfl*S#±LT, «B«k:±*y>rXo»* 
^rffll^S^d^LTtxfci^ 

[0 0 5 2] ±»LfcH*«!l8TH\ */f-f* 

[0053] 01 Hi. Ajc#sn^satJ*fei* 

^cfcoftBB»*»©aiL>G*«*Sfctt. 
S^rtT o TBHBktftKlOllffiR ^atB*rs^g^fcS 0 

*aaj-r*c: coBKtt*»ottnsRoAff 

*c#fiE*r SHSJcov >T«^ G ^^Ri6 S C k ^ RTftgtc a 
So CC0^3tcLT*^en/cS^GO3l^2^U-A 

[0 0 5 4] 

^B^-rsrc^tc, :7u-Aa#©s<ttt»*jiffiLT 
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[0ffi©ffitji&!M!] 

[04] /^->jt«»K«fesjt«»f¥o«fi*^-ria 

[0 5 ] :£Ka^M0lMl?BMStBlc iSttfffigOB 
[0 6 ] gmtt&HTfr LflHSfcfro fc*&©fij|l«yi© 

[0 7 ] ffirawau*^ fe*&©«»iaa© 
[0 8] BWfiBRfc^rsnsy-rxtiite-rsjSrffi©* 
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[0 9 ] n 2 <Dmmmm<omm> tf ai»/ < * 

[0 1 0] >J±«fflJt<l:Sit«i!lf^O«a«rS-r 

0Tfe5o 

[011] fii&©«{«*fflv"3«^©fttt0j&js-r0 
[£F^<D!£BJ3] 

1 0 

2 o tmsii 
2 2 »#a#*s^gp 

2 4 jJ-^-r^ Art/^a-itWgB 

2 6 sjg*!ui& 

2 8 S8><*->f&a«HISP 

3 0 SM/tjr->ttMff 

3 2 a^^-^ffrSfflSSB 
3 4 >Jti£tt 
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CH3] [06] 
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